GROUND STATE SOLUTION OF A NONCOOPERATIVE 
ELLIPTIC SYSTEM 



CYRIL JOEL BATKAM 



Abstract. In this paper, we study the existence of a ground state solution 
(that is a non trivial solution with least energy) of a noncooperative semilinear 
elliptic system on bounded domains. By using the generalized Nehari manifold 
method developed recently by Szulkin and Weth, we prove the existence of a 
ground state solution when the nonlinearity is subcritical and satisfies a weak 
superquadratic condition. 

1. Introduction 

In this paper, we are concerned with the following noncooperative elliptic system 

(— Au = F u (x, u, v), x e f2 
Av = F v (x,u,v), xett 
u = v = 0, x e dfl, 

where f2 is a bounded smooth domain in M. N and F u designates the partial deriv- 
ative with respect to u of the nonlinearity F : f2 x R 2 — * R. The solutions of such 
systems are steady state of reaction-diffusion systems which arise in many applica- 
tions such as chemistry, biology, geology, physics or ecology. It is well known (V) 
has variational structure, that is its solutions can be found as critical points of the 
following functional 

*(«,«) := £(^|V"| 2 - ||V^| 2 - F(x,u,v)) 

defined on Hq(ST) x H^fl) (that is the solutions of the equation <f>'(u,v) = 0, 
where $' is the Frechet derivative of $) . In this paper we will be interested in 
the existence of a ground state solution, that is a solution which minimizes the 
energy functional Let us recall that ground state solutions play an important 
role in applications. For instance, in the study of the formation of spacial patterns 
in various reaction-diffusion systems, the solutions of the system often converge to 
a ground state of a simplified semilinear elliptic system, as time tends to infinity 
(see0). 

In recent years the study of ground state solutions of elliptic equations and sys- 
tems has received a lot of interest, and some interesting results have been obtained 
(see for instance [H [31 [71 HI IS] and the references therein). In ( 7 , chapter 3) 
the authors presented the well known Nehari manifold method in a unified way, 
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which can be applied to find ground state solutions of the following cooperative 
elliptic system: 

—An = F u (x, u, v), x e il 
—Av=F v (x,u,v), x e 
u = v = 0, x e d£l. 

However, there appears to be no result in the noncooperative case. 

Let us introduce the precise assumptions on the nonlinearity F under which our 
problem is studied: 

(Fi) F e C l (fL xl 2 l) and F(x, 0) = for every x in IT. 

(F 2 ) \VF(x,u)\ < a(l + \u\p~ 1 ), for some p e (2,2*), x e il, u = (ui,u 2 ) e R 2 , 
where 2* := 2N/(N - 2) if N > 2 and 2* := oo if N = 1, 2. 

(F3) F(x,u) = o(|u| 2 ) as |u| — > 0, uniformly in x. 

(-F4) ^-fe^ — > 00 as I it j — > 00, uniformly in a;. 

(F 5 ) F(x, u) > and u ■ VF(x, u) > 2F(x, u), Vit e R 2 \{0}. 

(F 6 ) (w • Vf(i,ii))(«'i)) > 0. 

(F 7 ) |u| = \v\ => (F(x, u) = F(x, v) and v ■ VF(x, u) < u • VF(x, u)^ . 
If in addition u ¥= v, then i? • VF(x, u) < u • VF(x, it). 

(F 8 ) |u| ^ |u| and u • u ^ => w • VF(x, u) ^ u ■ VF(x, v). 

Where WF(x,u) = WF(x,Ui,U2) = (F ul (x,Ui,U2),F U2 (x,Ui,U2)) and u ■ v is the 
usual inner product in R 2 . A simple example of a nonlinearity satisfying these 
conditions is F(x, u) = f(x)\u\ p , where / > is of class C 1 on fL 
The main result of this paper is the following: 

Theorem 1. Under assumptions (i<\) — (F 8 ), (P) has a ground state solution. 

Let us point out that, since the energy functionnal $ associated to (V) is strongly 
indefinite (that is the negative and positive eigenspaces of its quadratic part are 
both infinite-dimensional), the set 

M := {ze H^(fl) x H^(fl)\z ± Oand (&(z),z) = 0} 

need not be closed (since inf^$ can be 0). Therefore Theorem [T] cannot be proved 
by using the usual Nehari manifold method (see [7], chapter 3 for a description 
and some applications of this method). To circumvent the difficulty posed by the 
strongly indefiniteness of $, we will use the generalized Nehari manifold method 
inspired by Pankov [I], and developed recently by Szulkin and Weth 0, which 
consists in a reduction into two steps. The materials used in this paper are also 
inspired by [7]. 

We organize the paper in the following way: In section 1, the method of the 
generalized Nehari manifold is briefly presented while in section 2 the existence of 
a ground state solution is proved. 
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2. The generalized Nehari manifold method 

Let X be a Hilbert space with norm || • || , and an orthogonal decomposition 
X = X + ©X~. We denote by S + the unit sphere in X + ; that is 

S + := {ueX + \ H| = l}. 

For u = u + + u~ e X, where u- e X-, we define 

X(u) := Ru © X- = Ru + © X~ and X(u) := R + u © X~ = © X~ (1) 

Let <I> be a C 1 — functional defined on X by 

*(«):= i||„+f-i|«-| a -/(«). 

We consider the following situation: 

(A x ) 7(0) = 0, \(l'(u),u) > > for all m ^ and I is weakly lower 

semicontinuous. 

(-A2) For each u> e there exists a unique nontrivial critical point of rh(w) 

of &\x< w y Moreover, m(w) is the unique global maximum of *&\x< w )' 

(A3) There exists 6 > such that ||m(w) + || > 5 for all w e X\X~, and for 
each compact subset /C c X\X~ there exists a constant C/c such that 
IrnHl < C K . 

We consider the following set introduced by Pankov [4]: 

M:= {ue X\X~ : ($'(w), it) = and (&(u),v) = Vw e X-}. 

Following Szulkin and Weth |7J, we will call .M the generalized Nehari mani- 
fold. 

Remark 2. By (Ai) M. contains all nontrivial critical points of $ and by (A2) 
X(uu) n M = {rh(w)} whenever w e X\X~ . 

In the following we consider the mappings: 

fh : X\X~ — > A4, w >— > m(w) and m := fh\g+. 

The following three results are due to A. Szulkin and T. Weth ([7J, Chapter 4), 
and we refer the reader to it for the proofs. 

Proposition 3. Assume that (A\), (A2) and (A3) are satisfied. Then: 

(a) fh is continuous. 

(b) m is a homeomorphism between S + and A4. 
Let 

§:I + \{0}-»l,%):=$(ra(i«)) and ^:=$| s+ . 
Proposition 4. Under assumptions (Ai), (A 2 ) and (A3), \t is of class C 1 and 

(*'(w),z) = ^fP(i'W,A for all w,zeX+, w * 0. 
\\ w \\ 

Before giving a consequence of the previous propositions, which is the main result 
of this section, let us recall some definitions. 

Definition 5. Let ip e C^X.K). 
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(1) A sequence (u n ) cz X is a Palais-Smale sequence (resp. a Palais-Smale 
sequence at level eel) for ip if (<p(u n )) is bounded (resp. ip(u n ) —* c) and 
ip'(u n ) — > as rt — > go. 

(2) We say that <p satisfies the Palais-Smale condition (resp. the Palais-Smale 
condition at level c) if every Palais-Smale sequence (resp. every Palais- 
Smale sequence at level c) has a convergent subsequence. 

Corollary 6. Assume that (Ai), (A 2 ) and (A 3 ) are satisfied. Then: 

(a) # e C 1 (S+,M) and 

(V'(w),z) = \\m(w) + \\($'(m(w)),z) for all z e T W (S), 

where T W (S) is the tangent space of S at w. 

(b) If (w n ) is a Palais-Smale sequence for then (m(w n )) is a Palais-Smale 
sequence for $. If (u n ) cz M is a bounded Palais-Smale sequence for 
then (m (w n )) is a Palais-Smale sequence for ty. 

(c) w is a critical point of \? if and only if m(w) is a nontrivial critical point 
of <&. Moreover, the corresponding critical values coincide and infs+^ = 
inf M <S>. 

3. Proof of the main result 

Let X := i?o(fi) x H&(Q) endowed with the norm ||(a,6)|| = (||Va||| 2(a) + 

||V6||^2(Q)) 2 , which by the Poincare inequality is equivalent to its usual norm. 
Define; 

X + := Hl(VL) x {0} and X~ := {0} x iJ^ft). 
Then for u = u + + u~ e X, we have 

*(«) = ^ll« + l| 2 -^ll«-|| 2 (2) 

where I(u) := l n F(x, u)dx. 
We recall that for u e X, 

X(u) := Ru©X~ = Ru + ®X~ and X(u) := R+u © X~ = R + u+ © X~ . (3) 

By a standard argument we have: 

Lemma 7. Under - (F 2 ), $ e C X (X,R) and 

(&(u),v) = J (\7u+ ■ Vv+ - Vu~ ■ Vv~ - Vv ■ VF(x, u)) . (4) 

Before giving the proof of the main theorem, we need some preliminary results. 
Lemma 8. Assume (Fi) and (F 5 ). Then (Ai) is satisfied. 

Proof. Clearly by (Fi) and (F 2 ) we have 1(0) = and (l'(u),u) > I(u) > 0, Vu ± 0. 
Let (u n ) cz X and eel such that u n u and I(u) c. By Rellich-Kondrachov 
theorem u n — » u in L 2 (fl) x L 2 (Q), and taking a subsequence if necessary we have 
u n (x) —* u(x) a.e on O. Since F is continuous, we conclude by applying Fatou's 
Lemma that I is weakly lower semicontinuous. □ 

Lemma 9. Under (F\), (F 3 ) — (Fa), (A 2 ) is satisfied. 
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Proof. (1) We first show that X(w) n M ¥= for any w e X\X~. 

Let w e X\X~. Then $ ^ on X(w)\Br for R large enough, where 
Br := {u e X | |u| < i?}. In fact, if this is not true then there exists 
a sequence (it„) c X(iu) such that ||u n || — » co and $(u„) > 0. Up to a 
subsequence we have v n = u n /\\u n \\ — ' v. By @ we have 

$(u„) 1 + 2 1 _ 2 f F(x,\\u n \\v n 



2" ™" 2' 



If w ^ we deduce, by using Fatou's Lemma and (-F4), that < -co; a 
contradiction. Consequently v = 0. Since = we may 

assume that w e S + . Now since I{u n ) 5= and 1 = ||^|| 2 + ||i^|| 2 , then 
necessarily u+ = s„ui ->■» 0. Hence there is r > such that ||v+|| = ||s n w;|| > 
r Vn. So ||v+|| = s n is bounded and bounded away from 0. But then, up to 
a subsequence, v£ — > sw, s > 0, which contradicts the fact that v n —- 0. 
By (F3), $(sw) = \s 2 + o(s 2 ) as s — > 0. Hence < supy, s $ < 00. Since 

<f> is weakly upper semicontinuous on X{w) and $ C on X(w) n X - , 
the supremum is attained at some point ito such that Uq 0. So mo is a 
nontrivial critical point of $\x( w ) an< ^ nence u o £ AL 
(2) Now we show that if u e M, then u is the unique global maximum of 
<%(„)• 

Let u e A4 and u + w e X(u) with w 0. By definition of we 
have u + w = (1 + s)tt + u, s ^ —1 and i> e X~ . By using the fact that 
s(| + l)it + (1 + s)v e X{u) we obtain 

$(it + w) - $(«) = -^||w|| 2 + 

J [( s (| + !) M + (! + s ) w ) ' V-Ffc, u) + F(a;, u) - F(x, u + u>)] . 
We define 5 on [—1, 00 [ by 

g(s) := (s(- + l)u + (1 + s)«) • VF(x, it) + F(», it) - u + w). 

Since u ^ 0, then in view of (F5) we have g{— 1) < 0. On the other hand 
we deduce from (F4) and (F5) that <?(s) — » — 00 as s — * 00. Assume that g 
attains its maximum at a point s e [— l,oo[, then 

ff '( s ) = ((1 + s )u + v) ■ VF(x, u)-u- VF{x, (1 + s)u + v) = 0. (5) 

Setting z = u + w = (l + s)u + v, on can easily verify that 

s 2 

g (s) = -{— + s + l)u- VF(x,u) + (1 + s)z ■ VF(x,u) + F(x,u) - F(x,z). 

It is then clear that if u ■ z < 0, then (Fe) implies g(s) < 0. Suppose that 
u ■ z > 0, then in view of ([S]), (F 8 ) implies |«| = \z\ and by (F7) we have 
F(x,u) = F(x,z) and z ■ WF(x,u) < u ■ VF(x,u) whenever w 0. This 
implies that g(s) < -^u ■ VF(x,u) 0. Hence $(u + w) < 

□ 



Lemma 10. Assume (F2) — (F$). Then (A3) is satisfied. 
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Proof. Clearly (F3) implies I'(u) = °(|u||) as |u| — » 0, which together with (j4i) 
imply that 

Ve > 0,3a > | Vm e X + ,\u\ <a^>I(w) < ~{l'(u),u) < < |||w|| 2 . 

Hence we can find p, rj > such that <&(w) 5= rj for any w e {u e X + \ \u\ = p}. 
By (A2), ^ rj for any w e X\X~. Since / ^ 0, we deduce from that 

|| 777,(1(7) + 1| S= \/2rj for any w e X\X~. 

Now let /C be a compact subset of X\X~. We want to show that there exists 
a constant C/c such that ||m(u;)|| < C/c, Vw e /C. Since m(w) = m(w + /\\w + \\) 
\fw e X\X~, we may assume that K, c £+. Suppose by contradiction that there 
exists a sequence (w n ) <= /C such that |m(iu n )|| — » 00. Since fh(w n ) e X(w„), we 
have m{w n ) = A„w„ + u n , with A„ 5= and w„ e X — . Since $(m(«; n )) > 0, 
\\w n \\ = 1 and / ^ 0, we deduce from ^ that X n 5= ||w„||. Hence A n — » 00, which 
implies |A n u>„ + u„| — ► 00 as n — > 00. By ([2j we have 

$(m(w„)) 1 _ 1 ||u„|| 2 _ f F(ar, A„w„ + «„) 

2 2 A2 J n 



< 



AH 2 2 AH Jo K 



1 1 IKII 2 f A„w» + |A„w» + «J 

2 2 AH J n |A„ Wri + f„| 2 AH 

1 f F(x,X n w n +v n ), l2 
o~ — u 7 12— ■ (*) 



Jn lAnWn+Wnl 1 

Since /C is compact we have, by taking a subsequence if necessary that w n — > w e S + 
and mj„ — > w a.e on f2. Clearly w ¥= 0. Then by using (F4) and Fatou's Lemma, we 
deduce from (*) that < -co; a contradiction. □ 

We need the following result: 
Lemma 11. Let 1 < q, r < 00 and G e C(fi xlxl) such that 

\G(x,a,b)\ <c(l+|a|* + |6|#). 

ITien Va,6 e G(-,a,b) e L r (fl) and the operator A : L q (fl) x L«(fi) -> 

Z/(f2), (a, 6) 1— > G(a;, a, b) is continuous. 

The proof of Lemma ITT1 follows the lines of the proof of Theorem A.2 in [5J and 
is omitted here. 

Lemma 12. Assume (F±) — (Fg). Then $ satisfies the Palais-Smale condition on 
M. 

Proof. Let (it„) c M. be a sequence such that < d for some d > and 

$'(«„) — » 0. We want to show that (u n ) has a convergent subsequence. 

Let us first show that (u n ) is bounded. 
If (it„) is not bounded, then up to a subsequence we have ||u n || — > 00. Define 
w„ := u„/||u n |. We easily deduce from ([2j) that 

£M = 1 +||2 _1 || 
IK|| 2 2 F "" 2 Fn " KP 

1,| +|,2 _ i||„- 112 f L. .a-^^VnlKI) 



2 11 nil 2 1 



f KP ^M^I . (**) 

Jn |w n ||M„lll 
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Since (v n ) is bounded we have, by taking a subsequence if necessary, v n — >■ v. If 
v 0, then by using one more time (F4) and Fatou's Lemma we obtain from (**) the 
contradiction ^ —00. Hence v = 0. Since > and I(u n ) > 0, © implies 

\\ v n II ^ II u n II- Hence we cannot have v+ — » (since ||w„|| = 1). There then exists 
a > such that, up to a subsequence, ||?;+|| 5= a Vn. It is clear that sw+ e X(u n ) 
Vs > 0. Then by (A 2 ) we have d > $(u„) > $(sv+) 5* ±s 2 a 2 -/(si>+) Vs > 0. Since 
u+ — ' 0, we deduce from the compactness of the embedding X °-» L p (f2) x L p (f2) 
that — > in L p (f2) x Now since by (F2) F satisfies the conditions of 

Lemma ITTI (with q = p and r = 1), we deduce that I(sv^) — » 0. It then follows 
that d > \s 2 a 2 Vs > 0. This gives another contradiction if we take s big enough. 
Hence (u n ) is bounded. 

By taking a subsequence if necessary we have u n u in X. It follows from the 
compactness of the embedding X L p (f2) x L p (f2) that u n —* u in L p (fl) x L p (fl). 
Now we easily obtain from ([2]) and ((4]): 

II^-m*! 2 = ±($'(u n )-$'(u),u±-u ± )± f ( U ±-u ± ).(V% U „)-VF(i,ii)). 

Jo 

Clearly ($'(ii n ) - $'(u), u± — it*) -» 0. By (F 2 ) the components of VF satisfy the 
conditions of Lemma [11] with q = p — 1 and r = , then by using the Holder 
inequality and Lemma [TT] we obtain j fi (it* — it- 

Consequently ti„ — > it. □ 

We also need the following consequence of the Ekeland variational principle: 

Lemma 13 ([5], Corollary 2.5). Let E be a Banach space and let ip e C 1 (i?,R) be 
bounded below. If tp satisfies the Palais- Smale condition at level 9 := infe ip, then 
there exists x e E such that tp'(x) = and 9 = tp(x). 

Proof of Theorem^ We already know from LemmaslSll^landfTUlthat (Ai ) — (Ai) 
are satisfied. By Corollary [6} (a) W e C 1 (S' + ,]R). 

Let us show that ^> satisfies the Palais-Smale condition on S + . 
Let (w n ) c S + be a Palais-Smale sequence for By Corollary [6} (b) (m(w n )) is 
a Palais-Smale sequence for $ on Ai. By Lemma [12] we have m(w n ) — » 10 up to 
a subsequence. Since m" 1 is continuous, it follows that w n — > m _1 (u;). Hence ^ 
satisfies the palais-Smale condition on S + . Particularly "J satisfies the Palais-Smale 
condition at level 9 = inig+ By Corollary [6]-(c) inf s + "A 7 = inf^n $ > and \I/ is 
bounded below. By Lemma [T3l inf g+ ^ is a critical value of ty. There then exists 
uq e S + such that inf 5+ ^ = ^(wo) and ^'(uo) = 0. It follows from Corollary 
|B]-(c) that m(uo) is a critical point of $ and $(m(it )) = infyK Hence m(uo) is a 
ground state solution for the equation $'(u) = 0. □ 
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